3d Lip-tracking for Audio-visual Speech Recognition in Real Applications

In this paper, we present a solution to the problem of tracking 3D information about the shape of lips from 2D picture of a speaker. We focus on lip-tracking of audio-visual speech recordings from the Czech in-vehicle audio-visual speech corpus (CIVAVC). The corpus consists of 4 h 40 min records of audiovisual speech of driver recorded in a car during driving in an usual traffic. In real conditions a head of a speaker (a car driver) can move and turn in various directions. To cope with these movements and to avoid recognition errors caused by changing 3D position of lips, our algorithm utilizes a 3D-model-based approach to the lip-tracking process. First, we present a method for creating and clustering the lip shape models. We derived 20 basic 3D models of the lip shape from real utterances of the speaker for representation of all lip movements. Next, we describe an algorithm for finding the shape of the lips in a picture using image processing. Further we present application of a distance function for choosing the best model for representation of the lip shape obtained by image processing. Finally we depict the results of the lip-tracking and its using for lip-reading.